May 19, 2017

Backtesting a Machine Learning Model

  • The model is trained over the lookback window, and tested out-of-sample on future data.

  • The length of the lookback window determines how quickly the model adapts to new information.

  • Backtesting allows determining the optimal length of the lookback window.

backtesting

Coin Flipping Model

coinflipping

  • Victor Haghani suggested a coin flipping model to illustrate the challenge of properly selecting a manager with skill, based on past performance.
  • We can select a manager from several managers, but only one of them has skill, and the remaining are without skill.
  • The skilled manager has a slightly greater probability of positive returns than negative ones, while the unskilled managers have a slightly greater probability of negative returns, so that the average performance of all the managers is zero.
  • If the probability of positive returns is equal to \(p > 0.5\), then the annual Sharpe ratio is equal to \(\sqrt{250}*(2p-1)\).
  • If the excess annual Sharpe ratio is equal to \(0.4\), then the probability of positive returns is equal to \((0.4/\sqrt{250}+1)/2 = 51.2\%\). ## Probability of Selecting a Biased Coin {.smaller}
  • We have a set of unbiased coins, except for a single biased one, with a \(60\%\) probability of heads.
  • We flip the coins simultaneously \(n\) times, and select the coin that produces the most heads.
  • What is the probability of selecting the biased coin, after flipping the coins simultaneously \(n\) times?

    ## Probability of Selecting a Skilled Manager {.smaller}
  • What is the probability of selecting the skilled manager (with an excess Sharpe ratio of \(0.4\)), from among two managers?
  • \(33\) years of data are needed to select the manager with skill, at \(95\%\) confidence!

    ## Selecting From Among Multiple Managers {.smaller}
  • In reality we must select from among multiple managers, any one of whom may out-perform purely by chance.

    ## Dynamic Investing With Multiple Managers {.smaller}
  • Dynamic strategy: at the end of each period, we switch to the best performing manager.

    backtesting
    ## Effect of Number of Managers {.smaller}
  • A greater number of managers decreases the out-of-sample strategy performance.

    ## Effect of Lookback Window Length {.smaller}
  • A longer lookback window increases the out-of-sample strategy performance.

    ## Simulating Managers with Time-dependent Skill {.smaller}
    ## Trend-following: Select Best Manager From Previous Period {.smaller}
    trend_following
    ## Ensemble: Select Top Two Managers From Previous Period {.smaller}
    ensemble
    ## Long-short Ensemble: Long Top Manager and Short Bottom Manager {.smaller}
    ensemble
    ## Thank You {.smaller}
  • Slide source is available here: \href{https://github.com/algoquant/presentations/blob/master/RFinance_2017.Rmd}}